[None][doc] add attention developer guide by QiJune · Pull Request #12693 · NVIDIA/TensorRT-LLM

QiJune · 2026-04-02T12:36:49Z

Summary by CodeRabbit

Documentation
- Added comprehensive developer guide documenting the attention stack architecture, including module organization, backend selection, metadata contracts, and KV-cache behavior.
- Updated documentation index with reference to the new guide.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

coderabbitai · 2026-04-02T12:39:17Z

📝 Walkthrough

Walkthrough

Documentation-only changes adding a new comprehensive developer guide for TRT-LLM's PyTorch attention module stack and updating the AGENTS.md reference index. The guide covers attention architecture, backend families, metadata contracts, KV-cache semantics, and testing guidelines.

Changes

Cohort / File(s)	Summary
Documentation Index `AGENTS.md`	Added reference row pointing to new attention developer guide with high-level description of covered topics.
New Attention Developer Guide `tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md`	Comprehensive new documentation covering attention module architecture, backend selection logic, metadata/runtime contracts, KV-cache ownership and decode semantics, MLA dispatch control flow, backend families and sparse registrations, testing pitfalls, anti-patterns, and evaluating new attention implementations.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	PR description is empty except for the template structure; critical sections like Description and Test Coverage are not filled out.	Fill in the Description section explaining what the attention developer guide covers and why it's needed. List specific test coverage or validation steps in the Test Coverage section.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding an attention developer guide documentation file.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md (1)
285-285: Reword “needs paged KV” to standard technical English.

At Line 285, consider rephrasing to “needs paged-KV support” or “requires paged KV” for clearer phrasing.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md` at line 285, Reword
the phrase "needs paged KV" at the sentence containing "If a source attention
implementation needs paged KV, chunked prefill," to standard technical English —
replace it with either "requires paged KV" or "requires paged-KV support" so the
sentence reads e.g. "If a source attention implementation requires paged-KV
support, chunked prefill," to improve clarity.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md`:
- Line 164: Update the hyphenation for three compound modifiers in
ATTENTION_DEVELOPER_GUIDE.md: replace "custom-op based" with "custom-op-based"
(occurrence at line where phrase appears), "cross attention" with
"cross-attention", and "KT-cache related" with "KT-cache-related"; ensure you
apply exact, case-sensitive replacements and run a quick repo-wide search for
these same unhyphenated phrases to fix any other occurrences for consistency.

---

Nitpick comments:
In `@tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md`:
- Line 285: Reword the phrase "needs paged KV" at the sentence containing "If a
source attention implementation needs paged KV, chunked prefill," to standard
technical English — replace it with either "requires paged KV" or "requires
paged-KV support" so the sentence reads e.g. "If a source attention
implementation requires paged-KV support, chunked prefill," to improve clarity.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a48b0f0f-ab23-48c6-b681-4097f0d22b92

📥 Commits

Reviewing files that changed from the base of the PR and between dbb1c8c and 81c61d9.

📒 Files selected for processing (2)

AGENTS.md
tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

QiJune · 2026-04-07T06:46:20Z

/bot run

tensorrt-cicd · 2026-04-07T06:52:02Z

PR_Github #42080 [ run ] triggered by Bot. Commit: 11d72f6 Link to invocation

tensorrt-cicd · 2026-04-07T07:23:14Z

PR_Github #42080 [ run ] completed with state FAILURE. Commit: 11d72f6
/LLM/main/L0_MergeRequest_PR pipeline #32919 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

QiJune · 2026-04-07T07:26:19Z

/bot run

tensorrt-cicd · 2026-04-07T07:33:04Z

PR_Github #42092 [ run ] triggered by Bot. Commit: 11d72f6 Link to invocation

tensorrt-cicd · 2026-04-07T08:08:09Z

PR_Github #42092 [ run ] completed with state FAILURE. Commit: 11d72f6
/LLM/main/L0_MergeRequest_PR pipeline #32931 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

QiJune · 2026-04-07T08:40:40Z

/bot run

tensorrt-cicd · 2026-04-07T08:46:30Z

PR_Github #42115 [ run ] triggered by Bot. Commit: 11d72f6 Link to invocation

tensorrt-cicd · 2026-04-07T10:37:08Z

PR_Github #42115 [ run ] completed with state SUCCESS. Commit: 11d72f6
/LLM/main/L0_MergeRequest_PR pipeline #32953 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

QiJune · 2026-04-07T11:05:56Z

/bot skip --comment "doc changes"

tensorrt-cicd · 2026-04-07T11:11:49Z

PR_Github #42137 [ skip ] triggered by Bot. Commit: 11d72f6 Link to invocation

tensorrt-cicd · 2026-04-07T11:20:38Z

PR_Github #42137 [ skip ] completed with state SUCCESS. Commit: 11d72f6
Skipping testing for commit 11d72f6

Link to invocation

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

add attention developer guide

81c61d9

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>

QiJune requested review from a team as code owners April 2, 2026 12:36

QiJune requested review from Shixiaowei02, laikhtewari and yuxianq April 2, 2026 12:36

github-actions Bot assigned QiJune Apr 2, 2026

QiJune requested review from kaiyux and lfr-0531 April 2, 2026 12:37

coderabbitai Bot reviewed Apr 2, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md Outdated

Shixiaowei02 approved these changes Apr 2, 2026

View reviewed changes